Papers with token-weighting methods
Token Weighting for Long-Range Language Modeling (2025.findings-naacl)
Copied to clipboard
| Challenge: | Many applications of large language models (LLMs) require long-context understanding, but models still struggle with such tasks. |
| Approach: | They propose token-weighting schemes that assign different weights to each training token in the loss, generalizing existing works. |
| Outcome: | The proposed methods compare confidences of a long-context and short-concept model and show that non-uniform loss weights improve the long-constability of LLMs. |